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FOREWORD 


The  work  reported  here  is  part  of  the  Human  Factors  Laboratory  program 
to  provide  automated  measurement  of  aircrew  performance.  Specifically, 
an  automated  system  for  debriefing  air  combat  performance  has  been 
developed  to  provide  feedback  that:  (1)  indicates  performance  trends 
over  time;  (2)  allows  comparison  of  performance  with  various  normative 
groups;  (3)  provides  diagnostic  results  for  individual  aircrews;  (4) 
formats  results  In  ways  to  reduce  unnecessary  complexity  for  the  In¬ 
structor  and  to  increase  the  ease  of  information  processing  by  the 
student  pilot;  (5)  relates  specifically  to  training  objectives .and 
(6)  Is  provided  more  rapidly  than  currently  possible.  The  preliminary 
version  of  the  automated  debrief  has  achieved  operational  acceptance. 

The  companion  volume  to  this  report  is,  "A  Problem  Definition  Study  of 
U.S.  Navy  Alr-to-AIr  Missile  Envelope  Recognition  Training  Interim  Report 
(Volume  II)."  Both  volumes  are  summarized  In  "Training  Improvements  for 
the  Tactical  Aircrew  Training  System  (TACTS):  Project  Summary  Report. 
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PREFACE 

Information  presented  in  this  Interim  Report  represents  partial  comple¬ 
tion  of  the  "ACMR  Training  Improvements  Project,"  sponsored  by  NAVTRA¬ 
EQUIPCEN  under  contract  N61339-78-C-0136.  The  Interim  Report  has  been 
divided  into  two  separate  volumes.  Volume  1,  Development  of  a  Computer 
Based  ACMR  Debrief  System,  is  an  unclassified  report  which  describes  the 
development  and  preliminary  design  for  an  ACMR  debrief  system.  Volume  II, 

A  Problem  Definition  Study  of  M.S.  Air-to-/-.ir  Missile  Envelope  Recognition 
Training,  is  a  classified  (Confidential)  report  which  addresses  current  fleet 
practices  and  potential  problem  aieas  in  missile  envelope  recognition  train¬ 
ing.  In  preparing  the  two  volumes  of  this  report  we  ha”e  chosen  to  use 
the  familiar  term  "Air  Combat  Maneuvering  Range  (ACMR)"  which  has  histor¬ 
ically  represented  the  Navy's  terminology  for  the  air  combat  instrumentation 
developed  by  Cubic  Corporation  Recently,  the  ACMR  has  been  re-designated 
by  COMNAVAIRSYSCOM  (06E)  as  the  "Tactical  Aircrew  Combat  Training 
System  (TACTS)."  We  will  adhere  to  the  new  designation  in  future  reports. 


1/2 


NAVTRAEQUIPCEN  78-C-0136-1 


TABLE  OF  CONTENTS 


Section  Page 


I  INTRODUCTION .  5 

Purpose .  5 

Background  .  . . 

II  TECHNICAL  APPROACH .  7 

Air  Combat  Maneuvering  Range  (ACMR) .  7 

Training  Design  Elements .  8 

ACMR  Training  Requirements .  . .  8 

ACMR  System  Components . . .  9 

ACMR  Training  Objectives .  11 

ACM  Sequence .  14 

Data  Collection  Requirements .  16 

Performance  Assessment  Methods  Development .  16 

Types  and  Levels  of  Performance  Criteria .  19 

Performance  Criteria  Standards . .  .  .  .  21 

Measurement  Framework .  22 

Summary  and  Discussion .  23 


III  DEBRIEF  DEVELOPMENT  AND  PRELIMINARY  DESIGN  ...  24 

vhe  Manual  System:  Data  Collection  and  Analysis  ...  24 

An  Automated  System:  Data  Collection  and  Analysis  ...  26 

Training  Analysis  for  Debrief:  Mission  Phases  and  Tasks  29 


Candidate  Measures  and  Data  Collection  Forms .  30 

Preliminary  Debrief  Data  Formats .  30 

Requirements  for  a  Computer  Based  System .  45 

Computer  Selection  and  Acquisition .  46 

IV  FUTURE  REQUIREMENTS .  50 

Criterion  Measurement  Research .  50 

Performance  Assessment  and  Appraisal  System  (PAAS)  .  .  50 

Future  Project  Plans .  53 

V  SUMMARY  AND  DISCUSSION .  55 

APPENDIX  A:  Preliminary  ACMR  Debrief  Check  List .  57 

APPENDIX  B:  Squadron  Performance  Comparisons .  65 

APPENDIX  C:  EMD  Sample  Data  Format  from  ACMR .  69 

APPENDIX  D:  Glossary  of  Air  Combat  Maneuvering  Terms .  71 


3 


NAVTRAEQLIPCEN  78-C-0136-1 


Figure 


2 

3 

4 

5 

6 
7 
C 
9 

10 

11 

B-l 


B-2 


B-3 


Table 

1 

2 

3 

4 

5 

6 


LIST  OF  ILLUSTRATIONS 

Page 


ACMR  Mission  Sequence .  15 

Criterion  Levels  and  Candidate  Measures .  20 

Flow  Chart  Depicting  ACMR  Data  Collection  and  Analysis 
Functions .  27 

Schematic  Representation  of  Proposed  ACMR  Retrieval 
System .  28 

Radar  Contact  Data  Format  Samples .  39 

Tally  Ho  Data  Format  Samples .  40 

Tactics  and  Maneuvers  Data  Format  Samples .  42 

Envelope  Recognition  Data  Format  Samples .  43 

Envelope  Recognition  Data  Format  Sample . .  .  44 

Relationship  Between  Criterion  Development  and  Training 
Improvement  Projects .  51 

Organization  of  the  Performance  and  Appraisal  System  .  .  54 

Search  and  Locate  Measures:  Moan  range  to  target  for 
radar  contact  and  Tally  Ho .  66 

Search  and  Locate  Measures:  Percent  radar  contacts  and 
Tally  Ho .  67 

Envelope  Recognition  Measures . 68 

LIST  OF  TABLES 


ACMR  Training  System  Components  and  Data  Bank  Taxon¬ 
omy  .  10 

Description  of  Primary  ACMR  Training  Objectives .  12 

Performance  Measures  for  Selected  Training  Objectives  .  .  17 

Analysis  of  In-Flight  Training  Objectives  and  Task  Dimen¬ 
sions .  31 

Candidate  Measures .  37 

***»i*y  Description  of  HP  45-T  Mini-Computer .  49 

4 


r 


i 

» 


1 

I 


i 


NAVTRAEQUIPCEN  78-C-0136-1 


SECTION  I 
INTRODUCTION 


Purpose 

This  section  of  the  Interim  Report  discusses  the  development  and  prelim¬ 
inary  design  of  a  computer  based  AQMR  debrief  system.  Such  a  system  as 
currently  envisioned  will  have  the  following  applications: 

1)  It  will  serve  as  a  vehicle  to  develop  and  test  statis¬ 
tically  summarized  ACMR  training  data  formats. 

Following  the  design  and  operational  validation  of 
candidate  training  data  formats,  selected  formats 
may  be  incorporated  on  the  ACMR  Data  Display 
System  (DDS)  for  use  in  providing  immediate  diag¬ 
nostic  training  feedback  to  operational  aircrews. 

2)  It  will  serve  as  a  baseline  or  prototype  system 
designed  to  provide  the  requisite  hardware  and 
software  components  to  store,  process,  retrieve 
and  display  cumulative  performance  trend  data 
required  for  fleet  personnel  to  evaluate  progress 
in  ACMR  training. 

Background 

Dunlap  and  Associates,  Inc.  scientists  have  been  involved  in  ACMR  train¬ 
ing  and  performance  measurement  for  the  past  four  and  one-half  years.  Their 
on-site  participal  .on  over  this  time  period  has  been  extensive.  Over  300  air-to- 
air  missions  and  post-mission  debriefs  have  been  observed  to  date.  Observations 
made  over  the  past  few  years  have  indicated  that  ACMR  training  missicr'*  and 
their  associated  debriefs  vary  considerably  in  training  emphasis,  and  in  the  con¬ 
tent  and  quality  of  information  presented  to  aircrews  under  instruction. 

Some  of  the  more  salient  ACMR  training  problems  observed  during  on-site 
participation  with  operational  aircrews  are  briefly  summarized  below: 
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1)  There  is  considerable  variation  in  the  type  and  quality 
of  ongoing  training  and  debrief  practices . 

2)  There  are  no  standardized  training  procedures  or 
structured  debrief  aids  available  to  fleet  personnel, 
beyond  the  customary  "operators  manual"  describing 
ACMR  functions. 

3)  The  ACMR  data  display  formats,  designed  to  present 
digital  and  graphic  information  depicting  ACM  engage¬ 
ment  geometry,  flight  instrument,  and  weapon  simulation 
data,  are  not  optimized  to  provide  summarized  statistical 
information  essential  for  immediate  training  feedback . 

4)  There  is  currently  no  user  operated  system  for 
storing,  analyzing,  retrieving  and  displaying 
cumulative  training  data  or  trend  analysis  infor¬ 
mation  required  to  evaluate  ACMR  t reining  progress 
on  a  longitudinal  basis. 

In  general,  current  ACMR  debriefs  have  been  found  to  be  highly  variable 
in  content,  unstandardized,  and  non-cumulative .  ACMR  performance  results, 
as  presently  used,  are  not  specifically  related  to  training  objectives  nor  are 
they  given  in  a  format  easily  understood  by  operational  users. 

Although  ACMR  presents  an  effective  application  of  advanced  instrumen¬ 
tation,  computing,  and  display  technology,  it  has  yet  to  benefit  from  a  con¬ 
comittant  application  of  state-of-the-art  training  methods.  The  development, 
design,  and  implementation  of  a  computer  based  ACMR  debrief  system  promises 
to  improve  substantially  the  utilization  of  ACMR  as  a  training  and  performance 
assessment  system.  A  systematic  approach  to  ACMR  training  and  performance 
assessment ,  exemplified  by  the  proposed  debrief  system ,  will  produce  the 
required  documentation  and  provide  users  with  the  necessary  framework  to 
accomplish  their  air  combat  training  mission. 
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SECTION  II 


TECHNICAL  APPROACH 


Air  Combat  Maneuvering 


The  ACMR  is  a  sophisticated  training  facility  acquired  by  the  Navy  and 
now  in  use  to  train  fighter  aircrews  in  air-to-air  combat.  The  system  is 
designed  to  train  aircrews  in  actual  combat  maneuvers  and  in  recognition  of 
weapoi  delivery  boundaries.  ACMR  provides  data  display  features  which 
greatly  enhance  air  combat  debriefs,  and  provide  a  rich  source  of  continuously 
recorded  quantitative  measures.  Some  of  the  capabilities  of  ACMR  include  the 
following: 1,2 

1)  Real-time  tracking  of  aircraft  engaged  in  air  combat 
training  in  a  specified  airspace, 

2)  Video  tape  playback  of  flight  history  data,  complete 
with  pictorial  display  of  the  air-to-air  engagement 
and  voice  transmissions, 

3)  Both  digital  and  graphic  hard-copy  printouts  of  flight 
instrument  data,  interaircraft  positions,  cockpit  view 
of  engaged  aircraft,  mission  data,  and 

4'  Computer  generated  estimates  of  weapon'  launch 
outoomes. 


^Brictson,  C.A.,  Ciavarelli,  A.P.  and  Jones,  T.N.  Development  ofAlrcrew 
Performance  Measures  for  the  Air  Combe  Maneuvering  Range  (ACMRJ  . 
Pensacola,  Florida:  Naval  Aerospace  Mt. uBil  Researcn  Laboratory,  Report 
No.  L 53001,  June  1977.  (CONFIDENTIAL) 


^rictson,  C.A.,  Ciavarelli,  A.P.,  Pettigrew,  K.W.  and  Young,  P.A. 
Performance  Assessment  Methods  and  Criteria  for  the  Air  Combat  Maneuverin 


ensacola,  Florida;  Naval 

erospace  Medical  itesearcn  laboratory,  special  Report  No.  78-4,  July  1978. 
(CONFIDENTIAL) 
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Training  Design  Elements 

A  thorough  consideration  to  the  rudiments  of  training  design  would  con¬ 
sider  such  elements  as: 

1)  ACMR  training  requirements  analysis, 

2)  Specification  of  training  objectives  and  their 

associated  operational  tasks, 

3)  Metnods  of  performance  assessment, 

4)  Instructor  selection  auU  training, 

5)  Adversary  pilot  training, 

6)  Training  data  and  display  formats, 

7)  Training  utilization  guides  which  include  a  training 
syllabus,  curriculum  design  and  lesson  plans,  and 

8)  Training  effectiveness  evaluation  techniques. 

At  the  outset  of  our  research  and  development  program,  we  have  sought 
to  maintain  our  level  of  effort  within  manageable  limits.  The  above  underlined 
training  design  elements  represent  those  which  could  be  addressed  during  the 
development  and  design  of  a  computer-based  ACMR  debrief  system.  The 
remaining  training  design  elements,  certainly  no  less  important,  can  be  added 
later  as  essential  "building  blocks"  required  to  complete  an  overall  systems 
approach  to  ACMR  training  design. 

ACMR  Training  Requirements 

The  Air  Combat  Maneuvering  Range  was  conceived  and  designed  to  meet 
the  following  operational  training  requirements: 

1)  Train  aircrews  in  all  aspects  of  ACM, 

2)  Provide  information  for  training  feedback  during 
debriefing, 

3)  Provide  quantitative  measures  cf  aircrew  proficiency 
and  combat  readiness,  and 

4)  Develop  maximum  ACM  proficiency  through  effective 
ACMR  use. 
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3  4 

These  requirements  were  derived  and  summarized  from  operational  doctrine  ' 
in  order  to  identify  the  overall  training  goals  that  the  ACMR  system  was 
expected  to  accomplish.  To  determine  how  well  ACMR  meets  these  requirements 
additional  analysis  was  necessary  to  more  fully  describe  the  ACMR  system  in 
terms  of  it's  training  and  measurement  capabilities. 

ACMR  System  Components 

For  purposes  of  analysis  and  description  the  ACMR  system  was  broken 


down  into  three  broad  categories:  personnel,  equipment  and  mission -related 
variables.  Those  categories  were  further  divided  into  six  major  system  com¬ 
ponents: 


Aircrews 

Personnel 

1. 

Instructors 

Aircraft 

Equipment 

Weapons 

j  • 

Mission 

Mission 

i. 

Environment 

Associated  with  each  component  are  certain  subsystem  elements  considered  as 

potential  sources  of  training  performance  variation.  Table  1  lists  the  six 
major  system  components  and  the  principal  variables  associated  with  each  com¬ 
ponent  . 

The  adequacy  of  ACMR  training  effect:  /eness  and  utilization  will  depend 
in  large  part  upon  understanding  the  relative  influence  and  interactions  of 
these  variables  on  aircrew  ACM  performance.  For  example,  ACM  performance 


0 

Commander  Operational  Test  and  Evaluation  Force.  Operational  Appraisal  of 
the  Air  Combat  Maneuvering  Range.  August  1974  (OPNAV  3930-11). 

4Naval  Air  Systems  Command,  ACMR  Evaluation  Summary:  AN/USQ-5r. 'T'). 
August  1975,  Report  No.  ACMR -1-1-76  1. 
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is  dependent  on  the  prior  experience  and  training  of  instructors  and  aircrews, 
type  aircraft  and  weapon,  and  mission  /environmental  variables.  Because  ACM 
performance  is  influenced  by  various  combinations  of  system  components  the 
description  and  breakdown  presented  in  Table  1  is  also  useful  as  a  data  bank 
taxonomy  and  reference  system.  Each  set  of  performance  data  collected  on 
the  ACMR  system  can  be  described  and  referenced  by  the  system  components 
identified  in  the  table.  ACM  performance  variations  can  thus  be  related  to 
specific  system  components  and,  if  desirable,  systematically  controlled  in  future 
training  sequences  to  assess  their  relative  influence  on  system  performance. 

The  identification,  organization  and  analysis  of  ACMR  variables  in  terms  of 
system  components  provides  the  necessary  framework  for  incorporating  ACM 
performance  assessment  methodology  into  a  computer-based  debrief  Bystem. 

ACMR  Training  Objectives 

The  Naval  Fighter  Weapons  School  (NAVF1TWEPSCOL)  located  at  Miramar 
NAS,  California,  is  the  primary  source  for  development,  instruction,  and 
dissemination  of  Navy  ACM  tactical  doctrine.  Training  emphasis  at  NAVFIT- 
WEPSCOL  is  placed  on  teaching  advanced  principles  of  air  combat  tactics  to 
fleet  experienced  aircrews.  As  such,  NAVFITWEPSCOL  instruction  provides 
a  certain  degree  of  commonality  concerning  the  content  of  tactical  doctrine 
and  how  ACM  missions  are  conducted  in  the  fleet. 

Project  personnel  established  contact  at  the  NAVFITWEPSCOL  in  order  to 
obtain  the  most  recent  information  on  Navy  ACM  instructional  objectives. 
Members  of  the  technical  staff  attended  classroom  training  at  NAVFITWEPSCOL, 
conducted  interviews  with  NAVFITWEPSCOL  instructors  and  reviewed  Navy 
ACM  tactical  documents.  An  outgrowth  of  those  activities  was  the  identifica¬ 
tion  and  description  of  primary  ACM(P)  training  objectives. 

Thirteen  ACMR  training  objectives  were  derived  through  analysis  of  the 
doctrinal  information  provided  by  NAVFITWEPSCOL.  Those  objectives  and 
their  corresponding  narrative  descriptions  are  presented  in  Table  2.  Prelim¬ 
inary  verification  of  the  13  ACMR  training  objectives,  in  terms  of  their  com¬ 
pleteness,  descriptions,  and  application  was  accomplished  by  working  sessions 
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TABLE  2.  DESCRIPTION  OF  PRIMARY  ACMR  TRAINING  OBJECTIVES 
Tracing  Objective*  Aircrew  Task  Description 


1.  Plan  of  Attack 


2.  Radar  Procedures 


3.  Look-Out  Procedures 


*  Mutual  Support 


5.  Section  Tactics 


8.  Tactics  and  Maneuvers 


7.  Energy  Maintenance 


8.  UHF  Use 


Establish  and  implement  an  effective  plan  of 
attack,  considering  adversary  capabilities, 
threat  area,  formation  and  fuel  requirements. 

Set  up  search  pattern,  i.e.,  beam  coverage  and 
scan  technique,  assign  section  responsibilities, 
and  operate  radar  equipment  to  establish  an” 
early  contact  and  to  enter  fight  with  best  tactical 
advantage . 

Utilize  weapon  system  displays  to  gain  early 
initial  tally  ho  and  maintain  look-out  responsi¬ 
bilities  to  keep  sight  of  all  adversaries  anc\ 
wingman. 

Provide  mutual  support  by  coordinating  section 
maneuvers,  communicating  intentions  and  adver¬ 
sary  position  data,  and  protecting  each  other's 
vulnerable  areas. 

Designate  and  maintain  free  fighter  and  engaged 
fighter  responsibilities  and  coordinate  fighter 
maneuvers  to  force  adversary  into  a  defensive 
position  for  a  kill. 

Use  recommended  offensive  and  defensive  tactics 
to  (1)  maneuver  fighter  to  adversary  lethal  zone, 
(2)  warp  envelope  away  from  attacking  adversary 
and  neutralize  adversary  threat. 

Maintain  energy  level  by  entering  fight  with 
good  energy  package,  unloading  when  possible, 
fighting  fast  with  minimum  arcing. 

Use  accurate  and  sufficient  UHF  commentary, 
including  calls  for  radar  status,  contact,  tally 
and  visual,  "six  clear,"  adversary  switches, 
position  information  and  transmission  of  inten¬ 
tions. 


£ 

Trailing  objectives  itemized  here  are  considered  to  be  categories  of  task 
dimensions  related  to  successful  ACM  performance. 
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TABLE  2.  DESCRIPTION  OF  PRIMARY  ACMR  TRAINING  OBJECTIVES  (Cont) 
Training  Objective  Aircrew  Task  Description 


9.  Cockpit  Coordination 


10.  Weapon  Selection  and 
Switchdogy 


11.  Envelope  Recognition 
and  Weapon  Fire 


12.  Bugout  Procedures 


13.  Debriefing  Procedures 


Provide  good  cockpit  coordination  through 
analysis  of  threat,  division  of  look-out  respon¬ 
sibilities,  and  discretionary  use  of  directive 
commentary . 

Select  a  weapon  appropriate  to  tactical  situation 
by  projecting  your  anticipated  aircraft  position 
relative  to  adversary,  planning  the  maneuvers 
required  to  reach  the  weapon  envelope  of  your 
choice,  and  setting  correct  switches  to  fire 
weapon  selected. 

Recognize  weapon  envelope  and  fire  within 
recommended  parameters  such  as  range,  angles 
off,  pointing,  closing  velocity,  and  oonsider 
other  relevant  factors  such  as  sun  angle,  wing- 
man  position,  clutter,  etc.  to  assure  successful 
launch  sequence. 

Disengage  or  bugout  when  required  to  meet 
projected  fuel  state  limits  and  other  possible 
tactical  disadvantages  which  may  seriously 
threaten  survival. 

Provide  knowledge  of  results  and  diagnostic 
training  feedback  during  debrief  in  regard 
to  successful  and/or  unsuccessful  completion 
of  training  objectives. 
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with  fleet  and  NAVFITWEPSCOL  personnel  and  through  briefings  with  CNO, 
COMNAVAIRPAC,  NAVFITWEPSCOL,  Fleet  Squadrons  and  COMFIT  command 
officers . 

The  identification  of  primary  training  objectives  provided  a  useful  frame 
of  reference  for  structuring  ACME  training  operations  and  deriving  candidate 
performance  measures  needed  to  assess  aircrew  task  performance  for  key 
training  objectives. 

An  analysis  of  training  objectives  and  related  aircrew  tasks  indicated  that 
all  of  the  itemized  training  objectives  were  not  readily  quantifiable,  nor  were 
the  individual  objectives  of  equal  importance  to  ACM  training.  As  part  of  the 
analysis,  it  was  necessary  to  select  certain  objectives  for  quantitative  evalua¬ 
tion  on  a  priority  basis.  Training  objectives  selected  for  study  met  at  least 
two  requirements:  1)  that  the  particular  objective  represent  an  important 
underlying  task  in  ACM  that  can  be  taught  using  the  range,  and  2)  that  the 
critical  task  components  associr  ,d  with  a  particular  objective  be  amenable  to 
quantification,  preferably  through  readily  accessible  ACMR  output  measures. 
The  primary  training  objectives  selected  for  evaluation  during  this  particular 
phase  of  debrief  development  are  itemized  below:  (see  Table  2  descriptions) 

•  Radar  Procedures 

•  Look-Out  Procedures 

•  Tactics  and  Maneuvers 

•  Envelope  Recognition 


ACM  Sequence 

Figure  1  depicts  a  simplified  ACMR  operational  sequence.  The  ACM 
sequence  provides  a  framework  for  collection  of  ACMR  in-flight  performance 
measures  at  key  points  in  an  air  combat  engagement.  As  indicated  in  Figure  1, 
there  is  a  correspondence  between  our  four  selected  training  objectives  and 
specific  points  in  the  ACMR  sequence.  The  points  selected  in  the  ACM 
sequence  provide  a  rational  analytic  structure  for  recording,  analyzing,  and 
reporting  training  data  for  immediate  debrief  and  for  later  review  of  perfor¬ 
mance  trends. 
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Data  Collection  Requirements 

Data  collection  procedures,  as  currently  cor  ducted,  call  for  recording 
ACMR  mt^sures  at  radar  contact,  tally  ho,  tirm  segment  between  tally  ho  end 
weapon  release,  and  during  missile  fire.  ACMK  rr assures  required  at  6ach 
point  in  the  air  combat  sequence  are  specified  in  Table  3.  included  in  Table  3 
is  a  listing  of  exercise  and  supplementary  data  required  for  describing  mission 
related  and  background  information. 

Performance  Assessment  Methods  Development 

The  emergence  of  ACMR,  with  its  aircraft  tracking  and  instrumentation 
capabilities,  has  provided  a  means  to  obtain  continuously  recorded  Inflight 
measures  of  system  performance  for  incorporation  to  the  ACMR  debrief  system. 
In  order  to  use  these  quantitative  outputs  for  aircrew  training  progress  and 
proficiency  evaluation  they  must  be: 

1)  Statistically  summarized  and  evaluated, 

2)  Qualified  with  respect  to  a  particular  combination 
of  system  components  and  operating  conditions, 

3)  Related  back  to  specific  training  objectives  and 
their  associated  task  elements,  and 

4)  Compared  with  ACMR  criteria  or  standards  of 
performance . 

Our  technical  approach  provides  the  framework  for  assessing  performance 
within  the  context  of  the  training  environment  and  across  ACM  variables  of 
interest. 
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TABLE  3.  PERFORMANCE  MEASURES  FOR  SELECTED  TRAINING  OBJECTIVES  (Cont'd) 
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Types  and  Levels  of  Performance  Criteria 

5 

In  a  previous  technical  report,  two  levels  of  criterion  measurement  were 
discussed.  Figure  2  illustrates  this  measurement  approach.  A  first  level  criter¬ 
ion  deals  specifically  with  the  terminal  aspect  of  air  combat  given  in  terms  of 
"engagement  outcomes"  (win,  loss,  and  draw  statistics).  The  use  of  engagement 
outcomes  is  important  for  evaluating  which  aircrew  won  or  lost  the  air  combat 
engagement . 

A  second  criterion  measurement  level  uses  measves  associated  with  "task 
accuracy'  to  assess  aircrew  performance.  Task  accuracy  measures  are  based 
on  interaircraft  tracking  data  from  ACMR.  The  importance  of  these  measures 
is  that  they  provide  a  wealth  of  information  related  to  successful  and  unsuccess¬ 
ful  aircrew  task  performance.  Information  regarding  how  a  particular  engage¬ 
ment  was  won  or  lost  is  provided  in  task  accuracy  measurement.  Examples  of 
task  accuracy  measures  include  continuously  recorded  range  and  angle-off-tail 
data  during  inflight  ACM  engagements.  Assessment  of  aircrew  proficiency  in 
ACM  entails  the  use  of  both  engagement  outcomes  and  task  accuracy  measures. 

During  the  course  of  measurement  research,  relationships  between  task 
accuracy  measures  (2nd  level  criteria,  or  proximal  measures)  and  engagement 
outcomes  (1st  level  criteria,  or  distal  measures)  will  be  investigated.  As  an 
example,  results  from  previous  research  have  already  established  the  follow:  g 

g 

relationships: 

•  Envelope  recognition  task  accuracy  scores, 
angle-off-tail,  interaircraft  range,  and  closing 
velocity,  correlate  highly  with  missile  fire 
success  (i.e. ,  ACMR  simulated  "kill,"  "no  kill" 
results) . 

•  The  likelihood  for  an  initial  virual  acquisition 
(Tally  Ho)  increases  with  a  successful  radar 
contact . 

g 

Ciavarelli,  A.P.,  Brictson,  C.A.  and  Young,  P.A.  Development  and  Applica¬ 
tion  of  Performance  Criteria  and  Aircrew  Assessment  Methods  for  the  Air  dornbat 
Maneuvering  Range  (ACMR)  (U).  Special  Report  79-5.  Pensacola,  Florida: 

Naval  Aerospace  Medical  Research  Laboratory,  September  1979.  (CONFIDENTIAL) 
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•  Success  on  search  and  locate  tasks  (radar  contact 
and  tally  ho)  results  in  an  increased  likelihood  of 
a  1st  shot  and  1st  kill  in  a  vectored  air  combat 
engagement . 

Analyses  and  results  &uch  as  those  presented  above  represent  a  part  of 
an  overall  research  effort  to  validate  the  measures  used  as  candidates  for  the 
debrief  system. 

Performance  Criteria  Standards 


4 


Candidate  measures  of  air  combat  performance  and  assessment  methods 
that  are  appropriate  for  debrief  purposes  will  be  selected  from  ongoing  ACMR 
criterion  research.  Some  important  considerations  in  this  selection  process 
are  discussed  in  this  section. 

ACMR  field  operational  measures  by  themselves  do  not  necessarily  repre¬ 
sent  valid  and  reliable  performance  criteria.  The  researcher  is  still  confronted 
with  the  unresolved  question,  "Which  measures  to  use,  and  why?"  Although 
ACMR  provides  a  rich  source  of  inflight  quantitative  measures ,  much  remains 
to  be  done  to  select  candidate  measures  which  can  be  meaningfully  related  to 
successful  performance  of  highly  multidimensional  air  combat  tasks.  In  addi¬ 
tion,  candidate  ACMR  measures  must  themselves  meet  stringent  measurement 
standards  prior  to  their  application  as  performance  criteria.  Brodsky 6 
presents  a  brief,  but  lucid,  set  of  measurement  standards,  which  are  listed 
in  an  adapted  form  below: 

Validity .  Measures  will  predict  performance  on  a  general 
set  of  tasks  which  contain  the  same  or  similar  elements. 

Reliability.  The  obtained  measures  can  be  reproduced 
under  the  same  or  similar  conditions. 


6Brodsky,  M.A.  The  use  of  full  scale  mission  simulation  for  the  assessment 
of  complex  operator  performance .  Human  Factors,  1967,  9(4),  341-348. 
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Sensitivity.  Measures  will  discriminate,  i.e.,  indicate 
changes  in  obtained  performance  with  the  application 
of  a  treatment. 

Operational  Utility.  The  use  of  a  task  measurement 
scheme  will  provide  data  which  are  applicable  to  pre¬ 
dict  performance  on  operational  tasks. 

To  these  standards  we  have  added  "user  acceptance^'  and  "data  avail¬ 
ability."  A  criterion  should  be  understood  and  acceptable  to  operaticnal  users. 
By  data  availability ,  it  is  meant  that  data  required  for  any  criterion  should  be 
in  sufficient  quantity  and  readily  available  to  allow  for  a  sound  statistical  base , 
i.e. ,  large  samples  and  easy  availability.  The  technical  approach  for  ACMR 
criterion  development  considers  these  particular  measurement  standards , 
together  with  a  systems  orientation,  as  an  essential  framework  for  deriving 
viable  air  combat  performance  criteria  for  later  incorporation  into  final  debrief 
design . 

Measurement  Framework 


Outputs  from  ongoing  criterion  research  have  been  used  to  construct  an 
integrated  measuremeut  framework  consisting  of  measures  of  air  combat  success 
for  selected  training  objectives  and  their  associated  task  elements.  Measures 
selected  for  final  application  in  the  ACMR  debrief  system  will  be  those  which 
satisfy  requirements  specified  in  the  criterion  research  phase  of  development , 
e.g.,  reliability,  validity. 

The  resulting  measurement  framework  will  address  critical  ACM  tasks 
and  acceptable  standards  of  performance.  For  example,  missile  launch  perfor¬ 
mance  assessment  can  be  established  on  the  basis  of  doctrine  missile  boundaries , 
e.g.,  thumbrule  envelopes,  as  an  acceptable  performance  standard. 
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Summary  and  Discussion 

The  development  and  design  of  a  computer-based  ACMR  debrief  system 
vill  roquire  at  a  minimum:  1)  extensive  knowledge  of  V«e  air  combat  mission, 

2)  specification  of  ACilR  training  objectives  and  associated  aircrew  tasks,  and 

3)  the  development  of  reliable  and  valid  performance  criteria  and  aircrew 
assessment  methods.  The  technical  approach  presented  thus  far  considered 
each  of  these  above  factor  ;.  Ey  and  large,  this  initial  conceptual  framework 
has  been  applied  during  our  previous  research  related  to  the  development  of 
ACMR  performan'**  criteria.  7 

This  fundamental  framework  will  serve  also  as  a  point  of  departure  and 
initial  foundation  for  a  proposed  ACMR  debrief  system .  The  following  sections 
in  Volume  I  of  this  interim  report  will  deal  directly  with  the  fuhctionrJ  require¬ 
ments  for  the  preliminary  ACMR  debrief  system,  beginning  with  the  ourrent 
manual  debrief  procedures  baseline  system)  and  ending  witn  the  preliminary 
design  features  of  an  automated  debrief  package  (a  computer-based  system). 


7 See  footnote  5,  page  18. 
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SECTION  III 

DEBRIEF  DEVELOPMENT  AND  PRELIMINARY  DESIGN 
The  Manual  System:  Data  Collection  and  Analysis 

System  Functions.  Certain  functions  have  to  be  performed  in  order  to 
obtain  meaningful  ACMR  data,  whether  automation  is  available  or  not.  Seven 
generic  functions  are  itemized  below,  followed  by  a  description  of  how  each  of 

Q 

these  functions  is  performed  on  the  manual,  or  baseline,  system. 

1)  Observation 

2)  Extraction 

3)  Transcription 

4)  Processing 

5)  Analysis 

6)  Review 

7)  Report 

Observation.  On-site  observation  of  ongoing  ACMR  training  has  been, 
and  will  continue  to  be,  an  essential  requirement  of  ensuring  the  quality  of 
performance  data  ootained.  In  some  instances  the  need  for  a  trained  observer 
is  mandatory.  For  example,  radar  contact  and  tally  ho  data  can  only  be  ob¬ 
tained  through  monitoring  voice  communications  during  ACMR  replay.  In 
addition,  presence  during  live  flights  and  aircrew  debriefings  is  required  to: 

1)  describe  and  define  ACMR  operating  conditions,  2)  qualify  unique  circum¬ 
stances  and  events,  3)  edit  and  remove  potential  bias  and  contamination  in 
source  data,  and  in  general,  to  understand  the  overall  operating  environment 
in  which  data  collection  has  occurred.  Quite  simply,  there  is  no  replacement 
for  the  trained  human  observer  in  performing  this  function. 


Sciavarelli,  Anthony  Evaluation  of  the  Readiness i  Estimation  System  (RES)  for 
ACMR  Data  Extraction  and  Analysis.  La  Jolla,  CA:  f)unlap  and  Associates, 
tnc.  March  1979.  (Kef  ared  for  Naval  Aerospace  Medical  Research  Laboratory, 

Contract  No.  N00812-78-C-8007. ) 

i 
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Extraction .  In  the  manual  data  collection  and  analysis  system,  data  are 
extracted  from  the  ACMR  display  system  through  use  of  the  hardcopy  printer. 
Content  of  the  data  collected,  therefore,  is  limited  to  the  formats  available  on 
the  ACMR  Display  and  Debrief  System  (DDS).  Other  sources  of  potentially 
useful  data,  such  as  time  history  measures,  and  "in"  envelope  indications  are 
not  available  for  analysis  in  the  present  manual  system. 

Transcription.  ACMR  raw  data  outputs  from  the  DDS  require  further 
transcription  to  an  analytic  framework  which  includes  organization  of  the  data 
by  squadron  or  unit,  aircrews,  aircraft  types,  weapon  types,  engagement  and 
weapon  fire  events,  and  other  pertinent  categories.  This  analytic  framework 
serves  as  the  basic  ACMR  performance  data  matrix  used  in  further  statistical 
analysis.  At  this  time,  data  entry  to  this  framework  is  a  manual  process. 

Processing.  Data  from  the  basic  matrix  are  coded  and  keypunched  for 
subsequent  computer  processing.  Once  data  are  on  IBM  cards,  efficient  use 
of  computer-based  statistical  and  analytic  programs  is  possible. 

Analysis.  Summarized  statistical  data,  after  computer  processing,  are 
reviewed  by  scientists  at  Dunlap  and  Associates,  Inc.  and  performance  trends 
are  studied.  In  some  cases  further  analysis  is  required.  Knowledge  of  air 
combat  tactics,  and  fleet  training  doctrine  is  essential  for  interpreting  results 
and  for  transforming  the  highly  technical  statistical  findings  into  operationally 
meaningful  data  formats. 

Review .  A  review  period  is  allowed  prior  to  final  reporting  in  order  to 
interact  with  operational  personnel  regarding  the  implications  of  significant  air 
combat  performance  trends.  Interaction  with  experts  in  the  operational  com¬ 
munity  has  been  an  indispensable  aid  toward  interpreting  variations  in  aircrew 
performance  observed. 


ii 
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Report .  Data  collection  procedures,  performance  assessment  methods, 
and  results  from  analysis  of  air  combat  performance  usually  have  been  docu¬ 
mented  as  a  final  research  report. 

The  manual,  or  baseline,  system  for  ACMR  data  extraction  and  analysis 
is  schematically  presented  in  Figure  3.  With  exception  of  statistical  data 
processing  accomplished  with  computer  assisted  techniques,  the  system  cur¬ 
rently  used  is  entirely  manual. 

The  following  section  of  this  document  presents  a  description  of  an  auto¬ 
mated  data  retrieval  system,  which  is  proposed  as  a  candidate  system  for  more 
efficient  data  extraction  and  analysis  necessary  for  immediate  training  feedback. 

An  Automated  System:  Data  Collection  and  Analysis 

An  important  consideration  in  the  final  design  of  an  ACMR  debrief  system 
will  be  the  development  of  automated  ACMR  data  extraction  methods.®  Figure 4 
depicts  a  proposed  conceptual  design  for  an  automated  ACMR  data  extraction 
system.  The  system  proposed  will  be  designed  to  ej.  tract  data  in  a  format 
tailored  to  specific  performance  measurement  requirements  and  to  fullfill  the 
need  to  present  timely  information  for  immediate  training  feedback  to  aircrews 
under  instruction. 

As  currently  envisioned,  the  retrieval  system  design  will  be  based  on  the 
use  of  a  special  purpose  disk  which  will  serve  to  call,  display,  and  print  ACMR 
data  in  a  prescribed  format.  The  retrieval  system  will  be  operated  during  or 
immediately  after  post-mission  debrief  in  order  to  edit  ACMR  data  with  perti¬ 
nent  debrief  information.  In  this  manner  summarized  statistical  data  related  to 
task  performance  in  air  combat  will  be  readily  available  to  aircrews  with  hard¬ 
copy  print-outs  serving  to  document  results  of  the  ACMR  missions  observed. 
Performance  data  obtained  as  hardcopy  results  can  be  further  transcribed  and 
analyzed  off-line  in  order  to  study  ACMR  performanca  trends  over  time  and 
repeated  engagements. 

®As  proposed  in  "Development  of  a  Performance  Measurement  and  Assessment 
System  for  ACMR"  (unsolicited  technical  proposal)  prepared  for  the  Naval 
Aerospace  Medical  Research  Laboratory,  Pensacola,  Florida,  May  1979. 
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Long-range  plans  for  more  efficient  retrieval  and  analysis  of  ACMR  data 
call  for  additional  hardware  interface  which  would  allow  a  direct  transcription 
of  ACMR  data  to  an  output  device  (i.e.,  magnetic  tape  or  disk).  Having 
data  in  an  appropriate  output  form  will  greatly  facilitate  off-line  data  pro¬ 
cessing  without  the  intermediate  step  of  manual  transcription.  If  possible, 
more  direct  means  (e.g.,  floppy  disk;  to  store,  retrieve,  and  input  data  to 
an  off-line  computer  system  will  be  incorporated  to  meet  our  overall  project 
objectives  to  provide  both  immediate  training  data  and  cumulative  trend  data 
to  uircrews  in  an  expeditious  manner. 

Requirements  for  the  design  of  an  automated  ACMR  retrieval  system 
include  the  following: 

e  Measurement  framework  specification 
e  Hardware  acquisition  (disk) 

e  Software  design  and  programming 

e  User  documentation 

e  Output  interface  (floppy  disk) 

e  Input  interface  (off-line  computer)* 

Training  Analysis  for  Debrief:  Mission  Phases  and  Tasks 

A  more  detailed  framework  for  analyzing  aircrew  task  requirements  is 
necessury  to  develop  an  ACMR  debrief  system.  Aircrew  tasks  were  broken 
into  discrete  components  and  allocated  across  the  following  mission  phases: 

•  Radar  search 

•  Radar  contact  and/or  lock-on 

•  Visual  search 

•  Initial  visual  acquisition  (tally  ho) 

e  Maneuver  to  envelope 

e  Weapon  release 

•  Reattack 

•  Bugout 


*HP  45-T  is  presently  available  for  off-line  analysis  and  debrief  data  format 
development . 
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Definitions  provided  in  Table  2,  previously  presented,  served  as  source 
material  for  analysis  of  aircrew  tasks  and  their  appropriate  allocation  across 
mission  phases  and  aircrew  members.  Table  4  presents  the  resulting  analysis 

« 

of  in-flight  training  objectives  and  task  elements  in  an  ACM  mission  framework. 

Candidate  Measures  and  Data  Collection  Forma 

Candidate  measures  for  selected  ACMR  phases  and  aircrew  tasks  are 
briefly  summarized  in  Table  5.  These  measures  correspond  to  more  detailed 
descriptions  presented  in  our  preliminary  debrief  data  colleotion  form.  A  ' 

i 

sample  data  collection  form,  together  with  aircrew  instructions,  is  presented 
in  Appendix  A .  To  aid  in  the  interpretation  of  air  combat  language,  a  glossary 
of  terms  is  provided  in  Appendix  D.  The  glossary  also  has  definitions  for 
the  candidate  measures  selected.  j 

The  format  for  collecting  deorief  data  is  still  under  development.  Future  ) 

project  plans  call  for:  1)  adding  other  training  objectives  and  measures,  jj 

2)  deriving  empirical  weighting  scores  based  on  the*  relationship  of  each  \ 

•I 

candidate  measure  to  ACM  success,  and  3)  automating  ACMR  data  collection 

and  analysis  (both  on-line  and  off-line).  • 

1 

I 

Preliminary  Debrief  Data  Formats  \ 

Data  formats  have  been  developed  for  the  following  ACMR  training  objec¬ 
tives: 

e  Radar  Procedures 
e  Look-out  Procedures 
e  Tactics  and  Maneuvers 
e  Envelope  Recognition 

These  preliminary  formats  are  designed  to  provide  statistically  summarized 
performance  data  '>r  both  immediate  training  feedback  (on-line  to  ACMR)  and 
cumulative  trend  analysis  (off-line  computer  based  system).  Provisions  have 
been  made  to  include  frequency  measures,  such  as  percent  successful  missile 
shots,  and  continuously  recorded  data  such  as  range  and  angle-off-tail  measures. 
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TABLE  4.  ANALYSIS  OF  IN-FLIGHT  TRAINING  OBJECTIVES  AND  TASK  DIMENSIONS  (Page  8  of  #) 
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TABLE  5.  CANDIDATE  MEASURES* 


MISSION  PHASE** 


CANDIDATE  MEASURES 


Radar  Search  and  Acquisition 


•  Radar  Opportunity  (yes,  no) 

•  Radar  Contact  (yes,  no) 

•  Radar  Lock  (yes,  no) 

•  Range  and  Time  of  Contaot 

•  Range  and  Time  of  Look-On 


Visual  Search  and  Acquisition 


Tally  Ho  (yes,  no) 

Range  and  Time  of  Tally  Ho 


i 


Maneuver  to  Envelone 


•  Time  and  Frequency  in  Offensive , 
Neutral ,  or  Defensive  State 

•  Fighter  First  Shot  Opportunity 
(yes,  no) 

•  Fighter  First  Kill  (yes,  no) 

•  Elapsed  Time  to  First  Shot , 

First  Kill 


Envelope  Recognition 


•  Interaircraft  Range 

•  Angle-Off-Tail  (AOT) 

•  Closing  Velocity  (Vc) 

•  Indicated  Airspeed  (IAS) 

•  Antenna  Train  Anglo  (ATA)*** 

•  Missile  Simulation  Result 
(kill  or  no-kill) 


*See  glossary  for  definitions. 


**  Search  and  Acquisition  phases  for  radar  and  visual  portion  of 
engagement  have  been  combined  to  simplify  this  presentation. 

***  Aircraft  pointing 
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Examples  of  these  data  formats  and  their  measurement  applications  are 
presented  below  for  each  of  the  above  listed  training  objectives. 

Radar  Procedures.  Tactical  doctrine  specifies  a  requirement  to  quickly 
locate  an  adversary  through  effective  use  of  onboard  radar  equipment.  An 
early  radar  contact  provides  aircrews  with  essential  position  data  required  to 
facilitate  visual  acquisition  of  an  adversary.  Also,  radar  contact  and  lock-on 
are  required  to  launch  certain  air-to-air  rissiles.  Figure  5  illustrates  two 
candidate  measures  for  assessing  radar  performance  based  on  success  rate 
and  ACMR  tracking  data.  Both  of  these  measures  have  demonstrated  sensi¬ 
tivity  in  discriminating  radar  performance  differences  between  some  operational 
units.  Performance  comparisons  presented  in  Appendix  B  illustrate  the  unit 
statistical  comparisons. 

Look-out  Procedures.  One  of  the  important  elements  of  "Look-out  Doc¬ 
trine"  specifies  obtaining  an  initial  visual  acquisition  (tally  ho)  as  early  in 
the  ACM  engagement  as  possible.  Obtaining  an  early  tally  ho  and  keeping 
sight  of  an  adversary  is  required  for  both  offensive  (maneuvering  to  opponents 
lethal  zone)  and  defensive  (protecting  your  owr  lethal  zone)  air  combat. 

Figure  0  presents  sample  data  formats  related  to  successful  visual  search  and 
acquisition  procedures.  Statistical  performance  comparisons  of  visual  acquisi¬ 
tion  between  squadron  units  are  also  presented  in  Appendix  B. 

Tactics  and  Maneuvers.  In  order  to  get  within  an  adversary  lethal  zone, 
a  pilot  must  first  maneuver  his  aircraft  to  arrive  at  the  proper  weapon  fire 
position.  Needless  to  say,  an  opponent  in  air  combat  has  the  same  goal.  The 
results  of  this  maneuvering  between  ACM  opponents  can  be  analyzed  in  terms 
of  offensive  (+),  defensive  (-),  and  neutral  (o)  engagement  states  defined 
in  terms  of  wingline  geometry  and  proximity  to  an  opponents  lethal  zone . ^ 0 


^See  footnote  8,  page  24. 
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A  wall  developed  ACMR  performance  measurement  Index  (PMI)  is  currently 
avalluble  for  incorporation  into  the  proposed  debrief  system.^  Figure  7  presents 
data  formats  related  to  tactics  and  maneuvers.  Candidate  measures  include 
percent  fighter  first  shots,  percent  fighter  first  kills,  and  continuous 
time-based  measures  usKa  the  PMI.  In  addition,  elapsed  time  to  first  Bhot, 
first  kill  and  survival  time  can  be  added  to  the  debrief  measurement  framework 
using  similar  data  formats. 

Envelope  Recognition.  In  order  to  defeat  an  opponent  during  an  air  com¬ 
bat  engagement  a  pilot  must  fire  his  weapon  within  the  boundaries  of  a  launch 
envelope.  As  a  rule,  weapon  launch  boundaries  are  specified  in  terms  of  inter¬ 
aircraft  geometry,  such  as  range  and  angular  limits.  Success  rates  and  weapon 
launch  accuracy  can  be  determined  in  terms  of  inter-aircraft  position  data  avail¬ 
able  on  ACMR.  Figure  8  presents  sample  data  formatB  for  selected  measures  of 
envelope  recognition  performance.  Success  rates  can  be  computed  from  ACMR 
simulation  outcomes  (kill,  no-kill)  or  from  doctrine  envelope  boundary  limits. 

For  example,  Figure  9  presents  an  array  of  rear-hemisphere  missile  fire  points 
in  a  polar  plot  format.  Each  of  these  fire  points  may  be  scored  by  ACMR  or 
through  application  of  doctrine  launch  envelopes  which  depict  range  and  angu¬ 
lar  limits.  More  information  regarding  precise  boundaries  of  weapon  fire  limits 
will  he  presented  in  a  later  volume  of  this  Interim  Report  (Volume  II). 

Unit  comparisons  for  envelope  recognition  performance  are  presented  in  Appen¬ 
dix  B . 

Other  Training  Objectives.  Measurement  scales  for  the  remaining  training 
objectives  (Table  t)  are  not  presently  available.  In  fact,  some  training  objec¬ 
tives  such  as  "mutual  support"  and  "cockpit  coordination"  may  be  impossible  to 
quantify  objectively  through  use  of  ACMR  outputs.  The  requirement  to  include 
such  "un measurable"  training  objectives  in  our  final  debrief  design  will  be 

^Simpson,  W.R.,  Oberle,  R.A.  The  Numerical  Analysis  of  Air  Combat  Engage- 
ments  Dominated  by  Maneuvering  PerformancTi  Patuxent,  ME>:  I'Javal  Air 
test  Center  ,  June  19 ?7. 
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explored  during  later  phases  of  debrief  development.  An  adequate  sampling 
of  debrief  content  may  call  for  inclusion  of  subjective  measurement  methods 
for  estimating  performance  effectiveness  on  those  ACM  tasks  which  are  not 
amenable  to  objective  measurement  approaches. 

On  the  other  hand,  the  training  objective,  "energy  maintenance"  can  be 
objectively  quantified.  Candidate  measures  related  to  energy  management  may 
be  available  through  use  of  the  energy  maneuverability  display  (EMD)  which 
is  currently  under  development.  A  sample  ACMR  display  format  for  the  EMD 
is  provided  in  Appendix  C. 


Requirements  for  a  Computer  Based  System 


The  previous  report  section  has  reviewed  some  of  the  data  formats  on 
selected  ACM  training  objectives  and  their  associated  tasks.  Performance 
measures  for  search  and  locate  tasks  (radar  and  tally  ho),  tactics  and  man¬ 
euvers.  and  envelope  recognition  are  currently  available.  Later  additions  for 
measuring  other  ACM  task  performance  also  are  planned.  The  capability  of 
comparing  these  measures  with  those  which  have  been  previously  documented, 
i.e,,  longitudinal  analysis,  will  permit  evaluation  of  statistical  trends  relative 
to  established  fleet  norms. 

During  the  course  ci  the  program  to  develop  ACMR  debrief  methods,  it 
has  become  increasingly  obvious  that  a  computer  based  system  will  be  required 
to  more  rapidly  process  and  display  performance  baaed  debrief  information. 

For  Navy  aircrews  undergoing  training,  this  means  that  performance  data  for 
critical  ACM  tasks  must  be  available  on  e  timely  basis,  and  must  appear  in  a 
format  understandable  by  aircrews  under  instruction. 

Information  required  during  debrief  and  in  later  training  reviews,  will 
range  from  simple  bar  charts  comparing  performance  on  single  engagements  to 
more  complex  presentations  such  as  fighter  position  at  weapon  fire  in  several 
dimensions  and  perspectives 

In  addition,  it  Is  important  that  squadron  training  officers  have  available 
a  rapid  retrieval  filing  system  of  "normative"  data  in  order  to  evaluate  progress 
in  training  over  time. 
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The  requirements  of  a  computer  based  system  are  twofold. 

1)  The  computer  provides  a  system  off-line  from  ACMR 
which  will  serve  as  a  vehicle  for  developing  and  test¬ 
ing  appropriate  ACMR  debrief  data  formats  (for  later 
incorporation  to  ACMR). 

2)  The  oomputer  serves  as  a  prototype  off-line  system 
for  storage,  processing,  and  display  of  ACMR  training 
data  required  for  trend  analysis  (longitudinal  review  of 
training  progress)  and  normative  data  bank  development. 

Computer  Selection  and  Acquisition 

Selection  Criteria.  It  is  essential  that  several  global  requirements  for 
selection  of  a  candidate  system  be  met.  These  particular  requirements  are 
itemized  and  discussed  below: 

1)  System  Access  and  Transportability 

2)  Store  Data  in  Retrievable  Files 

3)  Direct  Edit  of  Files 

4)  Cross  Tabs  and  Matrix  Manipulation 

5)  Statistical  Analysis  of  Data 

6)  Interactive  Graphics  Capability  (CRT  Display) 

7)  Hardcopy  Print-out 

8)  Secure  Data  Files 

System  Access  and  Transportability.  The  candidate  system  will  be 
used  initially  to  develop  and  test  various  performance  data  formats  required 
for  training  debrief  design.  Since  it  is  expected  that  the  process  of  develop¬ 
ing  and  testing  debrief  methods  will  involve  interaction  with  on-site  operational 
aircrews,  regular  daily  access  to  the  computer  based  system  will  be  a  require¬ 
ment.  Transportability  is  a  much  desired  design  feature  for  this  purpose. 

Data  Files.  Reviews  of  ACMR  training  data  frequently  entail  compar¬ 
isons  between  current  performance  trend  data  with  previously  documented 
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results.  Comparisons  of  this  type  are  essential  to  estimate  training  progress 
over  time  for  both  individual  aircrew  members  and  their  squadron  units.  In 
addition,  stored  information  related  to  various  combinations  of  aircrews,  air¬ 
craft  types,  weapon  systems,  mission  profiles  and  operating  conditions  must 
be  available  to  a  user  on  request  in  order  to  make  appropriate  performance 
trend  comparisons.  For  these  reasons  storage  and  retrieval  of  potentially 
large  data  files  is  considered  a  mandatory  system  requirement. 

Direot  Edit.  Incoming  ACMR  data,  collected  under  field  conditions, 
must  be  carefully  edited  to  establish  an  uncontaminated  data  base.  Direct 
editing  of  files  assures  expeditious  manipulation  of  incoming  data  for  file 
construction  and  for  retrieving  data  subsets  required  for  diagnostic  training 
feedback  of  performance  on  selected  aircrew  tasks.  Edit  routines  should 
include  capability  for  cross  tabs  and  matrix  manipulation  in  order  to  permit 
access  to  required  data  subsets. 

Statistical  Analysis.  The  foundation  of  the  developing  ACMR  debrief 
system  rests  on  comprehensive  statistical  analysis  and  review  of  ACMR  data. 

A  candidate  computer  system  ideally  would  have  a  menu  selection  of  both  des¬ 
criptive  and  inferential  statistical  programs  for  summarizing  and  testing  perfor¬ 
mance  trends. 

Interactive  Graphics  Capability.  The  application  of  state-of-the-art 
computer  graphics  is  a  user  oriented  requirement.  Initially,  interactive 
graphics  will  permit  experimentation  and  selection  of  graphic  presentation 
methods  which  have  high  operational  utility.  Data  displays  depictf  ;  perfor¬ 
mance  on  selected  aircrew  tasks  in  a  readily  understandable  format  will  be 
incorporated  in  the  final  debrief  system  design. 

Hardcopy  print-out  will  provide  the  capability  for  recording  results 
for  later  review  and  analysis  for  individual  aircrew  members  following  a  debrief, 
and  in  addition ,  will  serve  to  document  results  for  fleet  training  files  and 
longitudinal  anulysis. 
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Secure  Data  Files.  In  view  of  the  sensitive  nature  of  recording 
performance  capabilities  of  Naval  aircrews  and  the  potential  of  classified 
content,  it  will  be  necessary  to  have  secure  \CMR  data  files. 

Requirements  Summary.  Development  of  a  practical  system  for  ACMR 
debrief  will  require  a  computer  based  system  in  order  to  store,  process, 
retreive,  and  display  rapidly,  performance  trend  data  essential  for  timely 
training  feedback.  Although  preliminary  methods  have  been  developed  and 
tested  on  selected  air  combat  tasks,  further  refinement  and  implementation 
of  a  debrief  system  will  best  be  served  through  automation  methods.  To 
meet  this  need  for  automation  it  is  recommended  that  a  mini-computer  system 
be  used.  At  a  minimum,  such  a  system  should  be  readily  accessible,  pre¬ 
ferably  portable,  capable  of  editing,  storing,  and  retrieving  sizeable  files, 
and  feature  both  a  statistical  library  and  interactive  graphics  (CRT).  An 
additional  requirement  for  secure  data  files  is  recommended. 

Computer  Acquisition.  Candidate  computer  systems  were  reviewed^ 
and  evaluated  with  respect  to  the  above  selection  criteria.  On  the  basis  of 
this  evaluation,  the  Hewlett-Packard  (HP)  45-T  system  waB  selected.  The 
HP  45-T  System  is  a  state-of-the-art  desktop  computer  which  provides  rapid 
file  storage  and  retrieval,  editing  and  matrix  manipulation,  ready  software 
for  statistical  analysis,  interactive  graphics,  and  comes  complete  with  CRT 
display  and  built-in  hardcopy  output.  A  strong  point  in  favor  of  the  HP 
45-T  is  its  compact  design,  with  the  clear  advantage  of  transportability. 

The  HP  45-T  has  subsequently  been  delivered  (December  23,  1979)  and  is 
currently  undergoing  system  checkout,  and  software  inventory.  Application 
of  the  HP  45-T  system  to  computer  based  debrief  design  is  discussed  in  the 
next  section  of  this  report.  Table  6  lists  the  major  hardware  and  software 
features  of  the  HP  45-T  computer. 


^^Dunlap  and  Associates,  Inc.  Requirement  for  a  Computer  Based  ACMR 
Debrief  System.  Technical  Memorandum  234-4,  July  29,  1979. 
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TABLE  6.  SUMMARY  DESCRIPTION  OF  HP  45-T  MINI  COMPUTER 


Major  Hardware  Features 

•  187  K  byte  main  memory 

•  12"  CRT  display 

(80  character,  24  lines) 

•  Two  cassette  tape  drives 
(217  K  bytes) 

•  Keyboard  alpha  numeric  input 

•  Internal  thermal  printer 

•  Interactive  graphics  ROM 

•  Special  function  keys 


Software  Presently  Available 

•  Graphics  ROM 

•  General  Statistics 

•  Regression  analysis  (linear) 

•  Regression  analysis  (non-linear) 

•  Analysis  of  variance 
e  Statistical  graphics 

•  Supplementary  utility  programs 
(mass  storage  and  file  manipula¬ 
tion) 


System  Expansion  Potential 

•  480  K  byte  main  memory 

•  Mass  storage  ROM 

•  Flexible  or  hard  disk  drive 
s  High  speed  printer 

•  Card  reader 

•  A-D  converter  (digitizer) 
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SECTION  IV 

FUTURE  REQUIREMENTS 

Criterion  Measurement  Research 


We  have  defined  a  "criterion  measure"' J  as  an  acceptable  standard  by  which 
a  decision  or  judgment  may  be  made.  This  definition  presumes  that  the  candi¬ 
date  measures  selected  and  their  associated  performance  tolerances  have  satis¬ 
fied  both  statistical  and  operational  requirements  specified  in  an  earlier  section 
of  this  report  (e.g. ,  validity,  operational  acceptance).  Figure  10  summarizes 
the  relationship  between  ongoing  criterion  development  (measurement  research) 
and  debrief  development  (measurement  application).  Candidate  measures  them¬ 
selves  must  meet  high  standards  of  excellence.  For  this  reason,  the  quality  of 
the  criterion  research  has  been,  and  will  continue  to  be  the  foundation  for  an 
effective  debrief  system .  The  final  debrief  des.gn  will  rest  on  statistically  and 
operationally  verified  measures ;  with  emphasis  on  both  overall  ACM  performance 
assessment  and  diagnostic  appraisal  in  specific  task  areas . 

Performance  Assessment  and  Appraisal  System  (PAAS) 

Overview.  We  have  chosen  to  name  the  ACMR  debrief  system  the  "Performance 
Assessment  and  Appraisal  System  (PAAS)"  in  order  to  emphasize  its  application 
to  overall  ACM  performance  assessment  and  diagnostic  appraisal.  As  indicated 
previously,  PAAS  will  be  a  computer  based  system  which  uses  an  HP  45-T.  The 
HP  computer  provides  extensive  statistical  software  packages  which  will  have 
direct  application  for  debrief  design  (see  Table  6). 

In  addition,  it  will  be  necessary  to  develop  special  purpose  software  pro¬ 
grams  for  the  debrief  system  to  complement  the  HP  45-T  off-the-shelf  software. 
The  special  purpose  programs  will  serve  primarily  to:  1)  select  and  integrate 
off-the-shelf  statistical  routines  for  ACM  performance  review,  2)  include  user 


^3  See  footnote  1,  page  7 
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oriented  instructions  for  computer  operation  and  performance  evaluation ,  and 
3)  develop  a  coherent,  sequential  analysis  package  for  performance  assessment 
and  diagnostic  appraisal,  including  such  factors  as  data  taxonomy,  file  construc¬ 
tion,  editing  and  file  manipulation ,  and  menu  selection  of  various  programs  and 
statistical  analysis  methods. 

Software  Development.  The  PAAS  will  be  comprised  of  four  principal  programs: 
Standard  Operating  Procedures  (SOP),  Performance  Assessment  and  Combat 
Effectiveness  (PACE),  Numerical  Operational  Readiness  Measurement  (NORM), 
and  Training  Appraisal  and  Diagnostics  (TAD). 

Standard  Operating  Procedures  (SOP).  The  SOP  will  be  a  user  oriented 
program  which  provides  an  introduction  to  PAAS,  complete  operating  procedures, 
program  listing  and  description,  and  a  menu  selection  of  other  available  programs. 
Design  features  of  SOP  will  includ.;  the  use  of  branching  programming  methods  to 
facilitate  application  by  operational  users  with  varying  levels  of  operating  skill 
and  system  familiarity. 

Performance  Assessment  and  Combat  Effectiveness  (PACE).  The  PACE  pro¬ 
gram  will  be  designed  to  provide  a  menu  selection  of  ACM  performance  analysis 
and  assessment  methods  subroutines.  This  program  is  used  for  air  combat  effec¬ 
tiveness  across  various  training  objectives.  Provisions  are  made  in  the  PACE 
program  for  file  construction  and  manipulation,  including  a  data  taxonomy  cate¬ 
gorization  by  unit,  aircraft  type,  weapons,  aircrew  and  mission  variables. 
Analysis  subroutines  can  be  used  primarily  to  evaluate  performance  for  a  single 
ACM  detachment  or  training  period,  at  the  airwing,  squsdron  unit,  and  aircrew 
level  of  analysis.  Various  doctrine  referenced  criteria  (e.g.,  missile  envelope 
thumbruies)  may  be  selected  and  applied  for  conducting  performance  assessment 
against  fleet  training  standards. 

Numerical  Operational  Readiness  Measurement  (NORM) .  The  NORM 
software  package  will  permit  cumulative  storage  of  performance  data  files, 
unit  and  aircrew  training  records,  edit  and  file  update  routines,  and  rapid 
storage  and  retrieval  of  fleet,  unit,  and  individual  aircrew  data.  A  menu 
selection  of  summary  statistics  and  analysis  routines  will  be  avuilable  for 


NAVTRAEQUIPCEN  78-C-0136-1 


longitudinal  analysis  (using  a  normative  data  base)  of  training  progress 
against  norm  referenced  performance  criteria. 

Training  Appraisal  and  Diagnostics  (TAD).  The  TAD  program  will  pro¬ 
vide  a  capability  for  detailed  analysis  of  a  selected  training  objective  and  its 
associated  aircrew  task  performance.  Summarization  of  aircrew  strengths  and 
weaknesses,  together  with  recommended  remedial  training  will  be  provided. 

Software  Integration.  Figure  11  presents  the  organization  of  PAAS  in 
terms  of  its  principal  software  programs.  An  important  design  feature  of 
PAAS  will  be  the  integration  of  these  individual  programs  into  a  coherent, 
user  oriented  system.  Design  plans  call  for  an  interactive  system  having 
program  branches  which  allow  ready  access  and  cross-communication  between 
the  various  PAAS  programs  and  subroutines.  With  this  approach, a  user 
should  be  able  to  review  his  own  performance  and  compare  it  to  fleet  standards 
(PACE),  while  also  having  the  capability  to  call  normative  data  (NORM)  for 
performance  comparisons  to  fleet  levels.  In  addition,  TAD  will  give  training 
officers  the  capability  to  review  and  correct  training  deficiencies. 

Future  Project  Plans 

Immediate  project  plans  call  for  computer  programming,  software  devel¬ 
opment  and  integration  of  the  previously  described  PAAS  software  packages. 
System  operation,  proposed  programs,  and  data  formats  will  be  tested  in  inter¬ 
active  sessions  with  operational  aircrews.  A  demonstration  system  will  be 
designed  and  ready  for  review  in  approximately  six  months.  Long-range  pro¬ 
ject  plans  include  further  research  efforts  to  develop  candidate  measures  for 
other  air  combat  tasks,  e.g. ,  energy  maneuverability  and  to  incorporate  new 
data  formats  from  selected  measures  into  the  debrief  system.  Consideration 
will  be  given  to  meeting  high  performance  measurement  standards  (validity, 
reliability,  and  operational  utility) ,  and  for  reviewing  other  ongoing  training 
applications  of  performance  based  training  aids.  ^ 

^Mitchell,  D.R.  Trainee  monitoring,  performance  measuring,  briefing,  and 
debriefing.  In  1st  Interservice  Industry  Training  Equipment  Conference 
(NAVTRAEQUIPCEN  IH-316).  Orlando,  Florida:  Naval  Training  Equipment 
Center,  November  1979. 
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N  AVTRAEQU1PCEN  7H-C-0136-  1 
SECTION  V 

SUMMARY  AND  DISCUSSION 

Volume  1  of  thin  Interim  Report  has  presented  an  overview  of  the 

technical  approach  to  ACMR  debrief  development,  and  a  preliminary  debrief 
design  based  on  key  training  objectives  and  their  associated  aircrew  task 
measures.  The  requirements  for  a  computer  based  debrief  system  were  dis¬ 
cussed,  together  with  the  rationale  for  selection  of  the  HP  45-T  System.  Pre¬ 
liminary  data  formats  for  key  training  objectives  were  presented.  Finally, 

ACMR  debrief  software  packages  which  are  currently  under  development  were 
identified  and  described. 

The  preliminary  debrief  design  presented  in  this  report  represents  an 
outgrowth  of  an  ongoing  criterion  development  (measurement  rese.  rch)  pro¬ 
gram.  We  have  relied  heavily  on  the  extensive  experience  gained  on  ACMR 
and  our  knowledge  of  ACM  and  ACM  measurement  as  a  foundation  for  design¬ 
ing  a  debrief  system  that  provides  a  well  defined  measurement  approach. 

The  proposed  debrief  system  is  responsive  to  an  operational  requirement  for 
both  overall  performance  assessment  and  diagnostic  appraisal.  For  this  reason 
we  have  chosen  to  call  our  debriui  system  the  "Performance  Assessment  and 
Appraisal  System  (PAAS)." 

Future  plans  have  been  outlined,  and  center  chiefly  on  the  development 
of  u  demonstration  system  for  ACMR  debrief.  Further  development  of  criterion 
measures  across  remaining  training  objectives  will  be  necessary  to  complete  a 
final  debrief  design  such  that  it  represents  a  full  treatment  of  "all  aspect 
ACM"  performance  evaluation.  We  have  in  the  past,  and  propose  for  the  future, 
to  maintain  close  coordination  with  fleet  operational  personnel  in  order  to  deliver 
a  final  product  which  closely  mutches  their  ACMR  training  requirements. 
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APPENDIX  A 

PRELIMINARY  A-MR  DEBRIEF  CHECK  LIST 
FOR  MANUAL  SYSTEM 


•  AIRCREW  INSTRUCTION  SHEET 

•  DEBRIEF  QUESTIONNAIRE  FORM 


Debrio  flng 

1.  Operating  status  information  such  as: 

.  Radar 

.  VTAS/Seam 
.  Aircraft  and  Subsystems 

2.  Debrief  analysis: 

,  Conduct  debrief  by  ACM  mission  phase. 

.  Complete  debrief  checklist  for  each  fighter 
aircrew  and/or  section 


I 
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ACMR  PERFORMANCE  ASSESSMENT  AND  APPRAISAL  SYSTEM  (PAAS) 


OBJECTIVE 
Radar  Procedures 


MISSION  PHASE 
Radar  Search  and  Acquisition 


Critical  Tasks 

•  Designate  search -lock  responsibilities  for  section . Yes _ No 

•  Determine  eyeball  and  shooter  responsibilities . Yes _ No 

•  Operate  radar  equipment  to  establish  an  early  contact  .  .  .  Yes _ No 

•  Determine  adversary  formation  and  position  . . Yes _ No 

•  Set-up  for  initial  pass  using  offensive  split  and  to  enter 

fight  with  best  taatical  advantage . Yes _ No 

e  Overall  performance  for  radar  procedures1 2 * . S  OK  W 


Relative  Importance 


in  computing  total  score. 

Measurement  Scales 

•  Radar  Status . 

•  Radar  contact . Yes^ 

•  Radar  lock . Yes 


(weighting  coefficient)4 


_ Down  . .  .Time^ 

No _ A/C# _ Time_ 

No  A/C#  Time 


_Range_ 

Range 

Range 


Early  | _ 

32  or 
more 
□ 


31-26 

□ 


25-21 

□ 


20-16 

□ 


15-11 

□ 


Miles 


Instructor  Comments 


lStrong  (S),  OK,  weak  (W)  category  rating  scale  used  by  training  officers. 

2 

To  be  determined  from  results  of  ongoing  criterion  measurement  research. 
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ACME  PERFORMANCE  ASSESSMENT  AND  APPRAISAL  SYSTEM  (PAAS) 
OBJECTIVE  MISSION  PHASE 

3 

Lookout  Procedures  . . Visual  Search  and  Acquisition 


Critical  Tasks 

•  Keep  sight  of  each  other  . . Yea  No 

•  Identify  and  keep  sight  of  adversary (s) . Yes, _ No 

•  Re-acquire  adversary  after  losing  sight . Yes _ No 

e  Determine  direction  of  adversary  and  friendly  movement  .  .  Yes _ No 

e  Make  belly  checks  when  required . Yes _ No 

e  Overall  performance  for  lookout  procedures . S _ OK _ W_ 


Relative  Importance . . (weighting  coefficient) 

used  in  computing  total  score  , 


Measurement  Scales 

e  Tally  Ho  before  first  pass  (merge  plot) . Yes _ No _ 

e  Tally  Ho _ aircraft  in  fighter  section.  (How  many?) 

e  Tally  Ho  _ aircraft  in  bogey  section.  (How  many?) 

e  FirBt  Tally  Ho  made  by  A/C# _ on  A/C# _ :  Time _ Range _ 

•  Early , _ , _ I _ I _ + _ I _ I  Late 

7  or  6  5  4  3  2  lor 

more  □  □  □  □  □  less 

□  Miles  0 

Instructor  Comments 


3 


Search  and  acquisition  phases  have  been  combined  to  simplify  analysis. 
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ACMR  PERFORMANCE  ASSESSMENT  AND  APPRAISAL  SYSTEM  (PAAS) 


OBJECTIVE 


MISSION  PHASE 


Tactics  and  Maneuvers . Maneuver  to  Envelope 


Measurement  Scales 


•  Time  and  Frequency  in  Offensive-defensive  states^ 
e  Fighter  vs  Adversary  First  Shot  opportunity 

e  Fighter  vs  Adversary  First  Shot  time  elapsed 

•  Fighter  vs  Adversary  First  Kill  obtained 


e  Fighter  vs  Adversary  First  Kill  time  elapsed 
e  Scoring  method  for  Kill  determination: 


ACMR  Mode  Other 


Instructor  Comments 


‘These  measures  will  be  derived  from  Readiness  Estimation  System  (RES). 
The  RES  is  described  in  footnotes  8  and  9,  pages  24  and  26. 
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AC  MR  PERFORMANCE  ASSESSMENT  AND  APPRAISAL  SYSTEM  (PAAS) 


OBJECTIVE 


MISSION  PHASE 


Tactics  and  Maneuvers . Maneuver  to  Envelope 


Critioal  Tasks 


1)  Use  recommended  offensive  tactics  to: 
e  Predict  adversary  flight  path 
e  Make  adversary  predictable 
e  Maneuver  to  adversary  lethal  zone 
e  Obtain  a  quick  kill  (1  min.) 


2)  Use  recommended  defensive  tactics  when  required  to:  N/A _ Yes_ 

e  Warp  envelope  away  from  attacking  adversary  Yesa 

e  Mask /reduce  your  own  target  signature  (i.e. ,  come  Yes_ 
to  idle,  stay  in  clutter) 

e  Neutralize  remaining  threat  and  safely  disengage  Yes 


Relative  Importance 


(weighting  coefficient)  used  in  computing  total  score. 
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ACMR  PERFORMANCE  ASSESSMENT  AND  APPRAISAL  SYSTEM  (PAAS) 
OBJECTIVE  MISSION  PHASE 

Envelope  Recognition . . . Weapon  Release 


Satisfy  radar  lock  or  tone  as  required 

Yes_ 

_ No _ 

Meet  range  and  angle  requirements 

Yes_ 

_No _ 

Meet  pointing  requirements 

Yes_ 

_ No _ 

Satisfy  closing  velocity  requirements 

Yes_ 

_ No _ 

Meet  other  weapon  fire  restrictions,  i.e.,  clear  of 

clutter,*  and  wingman 

Yes_ 

_ No _ 

Relative  Importance . . (weighting  coefficient) 

used  in  computing  total  score. 


Measures 


Missile  Shots 

No.  Time  Shtr/Tgt  Missile  Range  APT  Result 
1-1 


Remarks 


1-2 

1-3 

1-4 

1-5 

1-6 

1-7 


1-8 


*  For  AIM  7  -  check  horizon  line  and  look  down 
For  AIM  9  -  check  sun  position  and  terrain  (when  necessary) 
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Instructor  Comments 


i 
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APPENDIX  B 


SQUADRON  PERFORMANCE  COMPARISONS* 


•  Radar  Contact  and  Tally  Ho 


•  Empirical  Envelopes 


j  1  i 


Performance  comparisons  using  doctrine  envelope  boundary  scoring  methods 
are  presented  in  Volume  II  of  this  Interim  Progress  Report. 
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CONTACT 
TALLY  HO 


PERCENT  ENGAGEMENTS 


1 


100n 


90h 


80H 


7(H 


6CH 


5CH 


40ju 


Figure  B-2. 
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n  radar 

L-1  CONTACT 
B  TALLY  HO 


Search  and  Locate  Measures:  Percent  radar  contacts  (R.C.) 
and  Tally  Ho  (T.H.)  comparing  two  F4  squadrons  (SQ  1  and 
2)  to  fleet  norm. 


I 

j 

i 

] 

4 

$ 
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Figure  B-3.  Envelope  Recognition  Measures:  Empirical  envelopes 

(5c  ±  lcr)  based  on  range  and  angle  measures,  comparing 
empirical  boundaries  for  two  F4  squadrons  (SQ  1  and  2) 
to  fleet  norm 
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i 
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APPENDIX  C 

j 

i 

1 

i 

i 

1 

EMD  SAMPLE  \ 

DATA  FORMAT  FROM  ACMR5 

$ 

i 

i 

! 

! 


^Pruitt,  V.R.  Energy  Management  Training  Aid  for  the  Navy's  Air  Combat 
Maneuvering  Range  (ACMR) .  (Technical  Report  N00123-78-C-1371) , 

St.  Louis,  MO:  Mct)onnel  Douglas  Corp.  ,  April  1979. 
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APPENDIX  D 

AIR  COMBAT  MANEUVERING  TERMS 


ACM 

ACM  State 


Acronym  for  Air  Combat  Maneuvering 


A  descriptor  of  the  ACM  situation  as  offensive, 
defensive,  and  neutral  positions 


AB 

AIM 

ALT 

AOA 

AOT 


After  Burner 


Air  intercept  Missile 


Altitude 


--  Angle  of  Attack 


Angle  Off  Tail;  angle  between  longitudinal  axis 
of  trget  and  line  of  t^ght  fxom  target  to  fighter 
in  wing  plane  of  target  aircraft,  measured  in 
degrees . 


ATA 


Angle  between  longitudinal  axis  of  fighter  and 
target  aircraft,  measured  in  degrees  (pointing 
angle). 


Bogey 

Bugout 


A  term  applied  to  an  ACM  opponent  (inspected 
unfriendly  or  adversary). 

Aircraft  leaving  arena  oi  engagement  or 
attempting  to  terminate  fight  (i.e.,  escape 
from  Bogey). 


Contact 


A  call  made  by  an  aircrew  member  (pilot  or  RIO) 
upon  obtaining  radar  contact  with  a  target. 


CRT 

DDS 


Cathode  Ray  Tube 


—  Display  and  Debriefing  System 
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Defensive 

HMD 

Energy 

Engaged  Fighter 

Envelope 

Fox  1 

Fox  2 

g 

GCI 

IAS 


--  An  engagement  state  in  which  a  particular  air¬ 
craft  is  in  a  threatened  position  according  to 
specific  mathematical  rules. 


—  Energy  Maneuverability  Display 


—  Kinetic  and  potential  energy  state  of  aircraft 
engaged  in  air-to-air  combat;  can  be  defined  in 
terms  of  IAS  and  cornering  trade-off  (kinetic), 
fuel  state  and  altitude  (potential). 


—  A  fighter  whose  primary  responsibility  is  to  kill 
or  control  bogey.  It  should  be  in  an  offensive 
position . 


—  Weapon  boundary  limits  within  which  a  missile 
or  guns  should  be  fired.  An  envelope  is  defined 
in  terms  of  distance  (range)  and  angles  off  tail 
(degrees)  between  shooter  aircraft  and  target. 


—  Call  made  by  aircrew  member  (usually  pilot) 
indicating  that  a  Sparrow  (AIM  7)  missile  has 
been  fired. 


—  Call  made  by  pilot  indicating  that  a  Sidewinder 
(AIM  9)  has  been  fired. 


—  Normal  Acceleration ,  measured  in  units  of 
g  =  32.2  ft/sec  . 


—  Ground  Control  Intercept 


—  Indicated  Airspeed;  airspeed  for  aircraft 
measured  in  knots. 
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Lock-on 

—  Electronically  locking  the  radar  system  on  a 
particular  target. 

Neutral 

--  An  engagement  state  in  which  a  particular  air¬ 
craft  is  in  neither  an  advantaged  or  disadvantaged 
state,  according  to  specific  mathematical  rules. 

Offensive 

--  An  engagement  state  in  which  a  particular  air¬ 
craft  is  threatening  an  opponent  (see  above 
defensive,  neutral). 

lvl 

--  An  engagement  involving  one  friendly  versus  (v) 
one  bogey  aircraft. 

PMI 

—  Performance  Measurement  Index 

Range 

--  Distunce  in  feet  or  nautical  miles  (n.m.)  between 
fighter  and  aircraft. 

RIO 

—  Radar  Intercept  Officer 

RTO 

--  Range  Training  Officer 

SEAM 

--  Sidewinder  Extended  Acquisition  Mode 

Section 

--  Two  aircraft  that  fight  as  coordinated  unit  in  an 
air-to-air  engagement. 

Tally  Ho 

--  A  call  made  by  an  aircrew  member  upon  obtaining 
visual  contact  with  a  target. 

2vl 

—  An  air-to-air  engagement  involving  two  friendly 
aircraft  versus  (v)  cne  bogey  aircraft. 

2v2 

—  An  air-to-air  engagement  involving  two  friendly 
aircraft  versus  (v)  two  bogey  aircraft. 
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UHF 

—  Ultra  High  Frequency  communication  channel 
for  radio  transmission  between  aircraft. 

Vc 

—  Closing  Velocity,  positive  or  negative,  between 
fighter  and  target  aircraft,  measured  in  knots. 

VTAS 

—  Visual  Target  Acquisition  System  (helmet  mounted 
gunsight  used  to  slave  and  point  weapon  seeker) . 

Wingman 

—  Second  aircraft  in  flight  section.  Also  referred  to 
as"wingy."  See  definition  for  "Section.” 

-  s' .  , 
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Commanding  Officer 
U.S.  Army  Engineer  Research  and 
Development  Laboratories 
Attn:  Library 
Ft.  Bel  voir,  VA  22060 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analyses 
Science  and  Technology  Division 
400  Army-Navy  Drive 
Arlington,  VA  22202 

Defense  Adv.  Research  Projects  Agency 
Information  Processing  Techniques  Ofc 
1400  Wilson  Blvd. 

Arlington,  VA  22209 

Defense  Adv.  Research  Projects  Agency 
Cybernetics  Technology  Office 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

Chief  of  Naval  Research 
Director,  Air  Programs 
Code  210 

Arlington,  VA  22217 


Dr.  Marshall  J.  Farr 
Director,  Personnel  and  Training 
Research  Program  (Code  458) 

Office  of  Naval  Research 
Arlington,  VA  22217 

Chief  of  Naval  Research 
Code  455 

800  N.  Quincy,. St. 

Arlington,  VA  22217 

Chief  of  Naval  Research 
Code  458 

800  N.  Quincy  St. 

Arlington,  VA  22217 

Mr.  Jerry  Maleckl 
Office  of  Naval  Research 
Code  455 

800  N.  Quincy  St. 

Arl ington,  VA  22217 

Dr.  Henry  M.  Halff 
Office  of  Naval  Research 
Code  458 

Arl ingtun ,  VA  22217 
Commanding  Officer 

Fleet  Combat  Training  Center  Atlantic 
Attn:  Mr.  Hartz,  Code  02A 
Dam  Neck 

Virginia  Beach,  VA  23461 

Commander 
Naval  Air  Force 

U.S.  Atlantic  Fleet  (Code  312  E) 

NAS  Norfolk 
Norfolk,  VA  23511 

Commanding  Officer 
FAS0TRAGRU1.ANT  (50) 

NAS  Norfolk  VA  23511 

Army  Training  Support  Center 
ATTSC-DS 

Ft.  Eustis,  VA  23604 

Conrad  Technical  Library 

U.S.  Army  Signal  Center  at  Ft.  Gordon 

Building  29807 

Ft.  Gordon,  GA  30905 
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Chief,  ARI  Field  Unit 
P.O.  Box  2086 
Attn:  Librarian 
Ft.  Bennlng,  GA  31905 

LSO  Training  Model  Manager 
Officer  In  Charge 
LSO  School 
Box  171 

NAS  Cecil  Field,  FL  32215 

Vision  Research  Division 
Code  32 

Naval  Aeromedlcal  Research  Lab 
Naval  Air  Station 
Pensacola,  FL  32508 

Chief  of  Naval  Education  and  Training 
Code  017 

NAS  Pensacola,  FL  32508 

Chief  of  Naval  Education  and  Training 
Code  N-2/CAPT  Bauchspies 
Pensacola,  FL  32508 

Commanding  Officer 
Naval  Education  Training  Program 
and  Development  Center 
Attn:  Technical  Library 
Pensacola,  FL  32509 

Chief  of  Naval  Education  and  Training 
Code  010 

Pensacola,  FL  32509 

Selection  and  Training  Division 
Department  of  Psychology 
Code  26 

Naval  Aerospace  Medical  Research  Lab 
Pensacola,  FL  32512 

TAWC  'TN 

Eg  1 1 »  AFP,  FL  32542 
Director 

Air  University  Library 
Maxwell  AFB ,  AL  36100 

Commanding  Officer,  U.S.  Army 
Avia  .Ion 

Air  Traffic  Control  School 

ATZQ-T-AT-ATC 

P.O.  Box  385 

Ft.  Rucker,  AL  36362  4  0f 


USAHEL/USAAVNC 

Attn:  DRXHE-FR  (Dr.  Hofmann) 

P.O.  Box  476 

Ft.  Rucker,  AL  36362 

Chief,  ARI  Field  Unit 

P.O.  Box  476 

Ft.  Rucker,  AL  36362 

Commanding  Officer 

Naval  Air  Technical  Training  Center 

Code  104,  Building  S- 54 

NAS  Memphis  (85) 

Millington,  TN  38054 

Chief  of  Naval  Technical  Training 

Code  0161 

NAS  Memphis  (85) 

Millington,  TN  38054 

CW02  Ray  Priest  (Code  7411) 

Naval  Air  Technical  Training  Center 
NAS  Memphis  (85) 

Millington,  TN  38054 

Mr.  Harold  A.  Kottmann 
ASD/YWE 

Wright  Patterson  AFB,  OH  45433 

Air  Force  Institute  Technology  Library 
Wright-Patterson  AFB,  OH  45433 

ASD/ENESS 

Attn:  R.  B.  Kuhnen 
Wright-Patterson  AFB,  OH  45433 

Air  Force  Human  Resources  Laboratory 
AFhRL/LR 

Logistics  Research  Division 
Wright-Patterson  AFB,  OH  45433 

6570  AMRL/HE 

Wright-Patterson  AFB,  OH  45433 

ASD/ENETC 

Mr.  R.  G.  Cameron 

Wright-Patterson  AFB,  OH  45433 

Commanding  Officer 
Naval  Hospital  Corps  School 
Great  Lakes,  IL  60088 
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Commandant 

U.S.  Army  Command  and  General  Staff 
College 

Library  Division 

Ft.  Leavenworth,  K S  66027 

Chief  of  Naval  Reserve 
Code  S- 331 1 

New  Orleans,  LA  70146 

Headquarters  34  Tactical  Airlift 
Training  Group/TTOI 
Little  Rock  AFB,  AK  72076 

Federal  Aviation  Administration 
AAC-954C 

Aeronautical  Center,  Flight  Standards 
Branch 

Oklahoma  City,  OK  73101 

Chief,  Methodology  and  Standards 
Staff 

Federal  Aviation  Administration 
Academy 

Aeronautical  Center,  AAC  -  914 
P.0.  Box  25082 
Oklahoma  City,  OK  73125 

Commandant 

U.S.  Arn\y  Field  Artillery  School 
Counterfire  Department 
Attn:  Eugene  C.  Rogers 
Ft.  Sill  ,  OK  73503 

Commandant 

U.S.  Army  Field  Artillery  School 
ATSF-TO-TS  (Mr.  Inman) 

Ft.  Sill  ,  OK  73503 

Headquarters 

Air  Training  Command,  XPTI 
Attn:  Mr.  Goldman 
Randolph  AFB,  TX  78148 

Commanding  Officer 
School  of  Aviation  Medicine 
Aeromed  Library 
Brocks  AFB 

San  Antonio,  TX  78200 
AFHRL/MP 

Brooks  AFB,  TX  78235 


Stimson  Library 
Academy  of  Health  Sciences 
U.S.  Army 

Attn:  Miss  H.  T.  Morrow,  DOCS  Librarian 
Ft.  Sam  Houston,  TX  78234 

U.S.  Air  Force  Human  Resources  Lab 
AFHRL-MPM 

Manpower  and  Personnel  Division 
Manpower  and  Force  Management  Systems 
Branch 

Brooks  AFB,  TX  78235 

U.S.  Air  Force  Human  Resources  Lab 
TSZ 

Brooks  AFB,  TX  78235 
AFHRL/PE 

Brooks  AFB,  TX  78235 

Chief  of  Naval  Air  Training 
Attn:  Code  3146  (LSO) 

NAS  Corpus  Christ! ,  TX  78419 

Chief  of  Naval  Air  Training 
ATTN:  Code  333 

NAS  Corpus  Christl,  TX  78419 
Superintendent 

U.S.  Air  Force  Academy  Library 
Code  DFSLB-D  . 

Denver,  CO  80840 

U.S.  Air  Force  Human  Resources  Lab 
AFHRL-OT  (Dr.  Rockway) 

Williams  AFB,  AZ  85224 

Commanding  Officer 
Human  Resources  Laboratory 
Operational  Training  Division 
Williams  AFB,  AZ  85224 

U.S.  Air  Force  Human  Resources  Lab 
AFHRL-FT  (Dr.  Edwards) 

Flying  Training  Division 
Williams  AFB,  AZ  85224 

Chief  of  Naval  Education  and  Training 
Liaison  Office 
Human  Resources  Laboratory 
Flying  Training  Division 
Williams  AFB,  AZ  85224 
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U.S.  Air  Force  Human  Resources  Lab 
AFHRL-OT 

Operational  Training  Division 
Williams  AFB,  AZ  85224 

AFHRL/OTO 

Luke  AFB,  AZ  85309 
Commanding  Officer 

Naval  Education  and  Training  Support 
Center,  Pacific 
Code  N5B  (Mr.  Rothenberg) 

San  Diego,  CA  92132 

Commander,  Naval  Air  Force 
U.S.  Pacific  Fleet  (Code  311  L) 

NAS  North  Island 
San  Diego,  CA  92135 

Commander,  Naval  Air  Force 
U.S.  Pacific  Fleet  (Code  342) 

NAS  North  Island 
San  Diego,  CA  92135 

Commander,  Naval  Air  Force 
U.S.  Pacific  Fleet  (Code  316) 

NAS  North  Island 
San  Diego,  CA  92135 

Commanding  Officer 
Fleet  Training  Center 
Attn:  Training  Department 
Naval  Station 
San  Diego,  CA  92136 

Commander,  Training  Command 
Attn:  Educational  Advisor 
U.S.  Pacific  Fleet 
San  Diego,  CA  92147 

Commanding  Officer 

Fleet  Combat  Training  Center,  Pacific 
Code  09A 

San  Diego,  CA  92147 
Commanding  Officer 

Fleet  Anti-Submarine  Warfare  Training 
Center,  Pacific 
Attn:  Code  001 
San  Diego,  CA  92147 

Commanding  Officer 

Naval  Health  Research  Center 

San  Diego,  CA  92152 
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Commander 

Naval  Electronics  Lab  Center 

Attn:  Library 

San  Diego,  CA  92152 

Navy  Personnel  Research  and  Development 
Center 

Attn:  M.  McDowell 
Library,  Code  P201L 
San  Diego,  CA  92152 

Commander 

Pacific  Missile  Test  Center 
Point  Mugu,  CA  93042 

LT  Wayne  R.  Helm 

Human  Factors,  Engineering 

Code  1226 

Point  Mugu,  CA  93042 

National  Aeronautical  and  Space 
Administration 

High  Speed  Research  Center  Library 
Edwards  AFB,  CA  93523 

Commanding  Officer 

Air  Force  Flight  Test  Center 

FTOTL 

Technical  Library  Branch 
Edwards  AFB,  CA  93523 

Commander 

Naval  Weapons  Center 
Code  3154  (Mr.  Bob  Curtis) 

China  Lake,  CA  93555 

Plans  Officer  Psychologist 
Ft.  Ord,  CA  93941 

National  Aeronautical  and  Space 
Adml nistratlon 

Ames  Research  Center  Aircraft 
Inspection  Branch 
Mall  Stop  211-5 
Moffett  Field,  CA  95050 


